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MOBILE DEVICE CONTROL SYSTEM 
BACKGROUND OF THE INVENTION 

TECHNICAL FIELD 

The present invention relates generally to robotics, and more specifically to mobile 
telepresence systems. 

BACKGROUND ART 

In robotic telepresence, a remotely controlled robot simulates the presence of a user. 
The overall experience for the user and the participants interacting with the robotic 
telepresence device is similar to videoconferencing, except that the user has a freedom of 
motion and control over the robot and video input that is not present in videoconferencing. 
The robot platform typically includes a camera, a display device, a motorized platform that 
includes batteries, a control computer, and a wireless computer network connection. 

SUMMARY OF THE INVENTION 

The present invention provides a system of mobile telepresencing, which includes 
moving a surrogate under wireless control, and autonomously moving the surrogate to regain 
wireless control when the wireless control is lost. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIGs. lA and IB show views of a Mutually-Lnmersive Mobile Telepresence System 
in accordance with an embodiment of the present invention; 

FIG. 2 is a plan view of a wheel configuration for a surrogate having six wheels; 

FIG. 3 is an isometric view of the surrogate using a steerable drive arrangement; 

FIG. 4 is a view of a differential drive system in accordance with another embodiment 
of the present invention; 

FIG. 5 is a view of a surrogate's forward motion path; 

FIG. 6 is a view of an approximation of a point that the surrogate can be expected to 
reach after initiating backtracking; and 
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FIG. 7 is a block diagram of a method of mobile device control in accordance with an 
embodiment of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

A Mutually-Immersive Mobile Telepresence System may have a mobile robot device 
5 of a humanoid as well as a non-humanoid shape, which is referred to as a "surrogate". A user 
is in a room that may show the surrogate's location and the surrogate may be located at a 
location, which may be a remote location. Video, audio, and motion control signals are 
transmitted between a user display and the surrogate by a wireless network, such as a wireless 
local area network or WLAN. The user is provided with views of the remote location on the 

10 user's display. These views look radially outward and 360-degrees around from the center of 
the surrogate in a surround view, which provides the user with the feeling of being present at 
the remote location. Likewise, the surrogate has display panels that project images of the 
user's head which provides the people or meeting participants at the remote location with the 
feeling that the user is present; i.e., the feeling of telepresence. 

15 The user sits or stands inside a display cube, with rear-projection surfaces on the front, 

back, sides, and optionally the ceiling, showing the surrogate's location. Since the goal is to 
be mutually immersive, live color video images of the user, centered on the user's head, are 
acquired from all four sides of the user's location for transmission to the surrogate's location 
concurrent with projection of live color video surround from the surrogate's location. At the 

20 same time, live color video surround is projected from the surrogate's location onto the four 
sides of the display cube surrounding the user. The user can control movement of the 
surrogate in real time over the wireless network. 

When using a wireless network, part of the remote location may not be covered by 
wireless services or may have dead zones. The range of WLAN access points are quite 

25 limited and large metal objects (such as HVAC ducts or elevators) can cause significant radio 
shadows. To date, WLANs have been deployed in unlicensed radio bands where power 
levels and signal strengths are limited. Furthermore, as higher and higher frequencies are 
utilized in order to achieve higher bandwidths, radio signals behave more like light rays and 
shadowing or blockage by metal objects becomes more pronounced. 

30 If the user of the surrogate drives into an area without wireless coverage, the surrogate 

can effectively become disconnected from the user. Since the user is generally not at the 
remote location with the surrogate, the user cannot push the surrogate into an area with 
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adequate wireless coverage. Moreover, since the connection to the surrogate from the user 
has been lost, the user cannot use the surrogate to ask people at the remote location to push 
the surrogate back into an area with adequate wireless coverage. Thus, it can be quite time 
consuming and inconvenient to regain connection to a surrogate that has moved outside areas 
5 with adequate wireless coverage. 

It has been discovered that, it is desirable that the surrogate may have additional 
capabilities of autonomous navigation in situations where the user cannot find an alternative 
path with better wireless coverage or the user desires to travel to a specific destination at a 
remote location where part or all of the path is blocked from wireless coverage. 

10 Referring now to FIGs. 1 A and IB, therein are shown views of a Mutually-Immersive 

Mobile Telepresence System 100, which includes a display cube 101 at a user's location 104 
and a surrogate 106 at a surrogate's location 108. The surrogate 106 is connected to the 
display cube 101 via a high-speed network 1 10. 

The surrogate 106 has a surrogate's head 112 including a number of head display 

15 panels 1 14, such as four liquid crystal display (LCD) panels. One or more cameras 116, such 
as four surrogate's cameras 116-1 through 4, are positioned in the comers of the surrogate's 
head 112. The surrogate's cameras 116 view and capture 360-degree surround live video at 
the surrogate's location 108 for display on the display cube 101. One or more microphones, 
such as four directional surrogate's microphones 1 18, are positioned in the top comers of the 

20 surrogate's head 112 to capture sounds 360 degrees around the surrogate 106. One or more 
speakers, such as the four surrogate's speakers 120 are also positioned in the bottom corners 
of the surrogate's head 1 12 to provide directional audio of the user's voice. 

The surrogate 106 contains a surrogate's computer/transceiver system 122 connecting 
the surrogate's cameras 116, the surrogate's microphones 118, and the surrogate's speakers 

25 120 with the display cube 101 for a user 124. The surrogate's computer/transceiver system 
122 also receive live video views of the user's head 126 from user's camera arrays 128 at the 
four comers of the display cube 101 and display the live video views on the head display 
panels 1 14 in the surrogate's head 1 12. 

The display cube 101 at the user's location 104 receives the live video and audio 

30 signals at user's computer/transceiver systems 130. The video view from the four surrogate's 
cameras 116 in the surrogate's head 112 are projected on projection screens 102 of the 
display cube 101 by four user's projectors 132. 
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User's speakers 134 are mounted above and below each projection screen 102. By 
powering each pair of user's speakers 134 with equal volume signals, the sound appears to 
come from the center of each of the projection screens 102 and provides directional audio or 
hearing of one or more participants from the four surrogate's microphones 1 18. 
5 The user's computer/transceiver systems 130, which can be placed in an adjacent 

room (for sound isolation purposes), drive the user's speakers 134 with audio information 
transmitted from the surrogate 106 at the surrogate's location 108. 

The images on the projection screens 102 are presented "life size". When the user's 
head 126 is centered in the display cube 101, the angle subtended by images on the projection 

10 screens 102 is roughly the same angle it would be at if the user 124 was actually at the 
surrogate's location 108 viewing it personally. 

To have full surrogate mobility, the surrogate 106 can have remote translation and 
remote rotation capabilities. The term "translation" herein means linear movement of the 
surrogate 106, and the term "rotation" herein means turning movement of the surrogate 106. 

15 When the user 124 desires to change body orientation with respect to the surrogate's 

location 108, the user 124 may do so by turning at the user's location 104 and having the 
surrogate 106 remain stationary. The head display panels 114 on the surrogate 106 show the 
user's head 126 turning to face the desired direction without movement or a rotation of the 
surrogate 106. In another embodiment, the surrogate 106 would be capable of turning, 

20 The surrogate 106 has a surrogate's body 140, which is rotationally (circularly) 

symmetric and has no front, back, or sides (i.e., the base and body of the surrogate 106 are 
cylindrical). Furthermore, the surrogate 106 uses a mechanical drive system 144 that can 
travel in any translational direction without a need for rotation of the surrogate's body 140. 

To prevent collisions between the surrogate 106 and remote objects while the user 124 

25 is driving the surrogate 106, the surrogate 106 is equipped with distance-measuring and 
collision-avoidance devices 145, such as ultrasonic and/or infrared detectors, that can 
override the commands from the user 124 that can otherwise result in damage or unsafe 
operation. 

In the present invention, forward movement is under real-time wireless user control 
30 while backtracking and predetermined routing is performed autonomously under the control 
of the surrogate's computer/transceiver system 122. Since the surrogate 106 is more prone to 
problems when it is traveling autonomously, an autonomous control waming 147 is provided 
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to the people at the remote location by blinking a small yellow light and/or producing a quiet 
"backup beeper" type of audio warning (e.g. 1 kHz sine wave, 50% duty cycle per second). 

Referring now to FIG. 2, therein is shown a plan view of a wheel configuration for a 
surrogate 106 having six wheels 200 set to go in a projected direction 202. 
5 The steering angle of the wheels 200 in the base of the surrogate 106 is only important 

when the surrogate 106 is in translational motion. Thus, to save power and reduce noise and 
vibration while the surrogate 106 is at a fixed location, the steering angle of the wheels 200 
may be left unchanged until the surrogate begins translational motion (i.e., a "lazy" update 
policy). 

10 Referring now to FIG. 3, therein is shown an isometric view of the surrogate 106 

using a steerable drive arrangement 300 for accomplishing translation in any direction at the 

surrogate's location 108 without rotation of the surrogate 106 itself. 

The steerable drive arrangement 300 has steering gears 300-1 through 6 for the six 

wheels 200 (four shown). This provides enhanced stability over previous bases with three 
15 wheels. The wheels 200 are made large enough to surmount the largest vertical discontinuity 

allowed by the Americans with Disabilities Act. 

Each wheel 200 can be steered through the use of a steering sprocket 304 and can be 

driven through a drive sprocket 306. All the wheels 200 are tied together by steering and 

drive chains so that they always face the same direction and rotate at the same speed. The 
20 wheels 200 are housed inside a cylindrical fairing (not shown), so that they are not easily 

visible to participants at the surrogate's location 108. 

The user 124 using the surrogate 106 appears to glide over the floor in the direction 

that the video images of the user 124 are facing. When the user 124 turns and moves in 

another direction, the video images of the user 124 shows the rotation of the user's head 126 
25 to a new orientation on the head display panels 114 of the surrogate 106. However, only the 

wheels 200 of the surrogate 106 under the fairing rotate and not the surrogate's body 140. 

Rotation of the wheels 200 about their steering axis encounters much less inertia than rotating 

the whole surrogate's body 140, and also uses less energy. 

A support 314 hangs down from the center of the lower plate for supporting a battery 
30 (not shown). 

In one embodiment of the present invention, distance-measuring and collision- 
avoidance hardware 145 is used to assist the user 124 in driving the surrogate 106 during 
normal operation. It is also used to help operate the surrogate 106 autonomously to "follow 
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its footsteps" back into an area with adequate wireless coverage. Once in the area with 
adequate wireless coverage, the user 124 can regain control over the surrogate 106. The 
surrogate 106 only needs to back up by a limited amount, since the communication loss can 
be detected within several seconds (at most) of its occurrence. 
5 If the surrogate 106 cannot reacquire communication with the user 124 after reversing 

a specific distance, it halts and waits. This is because the surrogate 106 has retumed to an 
area where it previously had good communication with the user 124, and the inability to 
restore communication may be due to other causes such as a failure at some other point in the 
network. 

10 In some situations, the surrogate 106 may not be able to return along the path 

previously taken due to the appearance of obstacles; e.g., a door has been closed behind it. In 
this case, the surrogate 106 will temporarily suspend backtracking, and try to keep a specified 
distance between the surrogate 106 and surrounding obstacles while remaining on the path by 
using the distance-measuring and collision-avoidance hardware 145 of FIG. 1. 

15 It has been discovered that one common feature of many WLANs is that the supported 

bandwidth varies with distance and signal path quality to the access point. At significant 
distances from the access point, or if significant obstructions occur in the path to the access 
point for the surrogate 106, the communication can downshift to a much lower bandwidth. 
For example, systems complying with the 802. 11 g standard support data rates of 54, 48, 36, 

20 24, 18, 12, 9, and 6 Mb/sec. If the surrogate 106 needs a data rate greater than the minimum 
data rate for normal operation (e.g., 10 Mb/sec), networking failures will occur when the 
surrogate 106 moves into an area supporting only a lower data rate. Such failures can consist 
of dropped packets or rapidly increasing latency, depending on the protocols and hardware 
being used. When the percentage of packets lost exceeds a threshold percentage or packet 

25 transmission delays exceed a time threshold, the surrogate 106 can treat the connection as 
effectively being lost and initiate backtracking. 

It has also been discovered that traffic from other wireless clients can overload an 
access point being used by the surrogate 106. This can cause loss of packets or excessive 
packet delays. If these problems are transient (e.g., lasting less than several seconds), the 

30 surrogate 106 will not need to initiate backtracking. The automatic backtracking begins when 
a networking problem persists for several seconds or more. 
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During normal operation, the surrogate 106 keeps track of its relative location in real- 
time as the user 124 is driving it by monitoring position encoders on the wheels 200 of the 
surrogate 106. This type of navigation is called "dead reckoning**. 

In dead reckoning, the direction and distance traveled is summed with the previous 

5 distances and directions to compute a new position; i.e., vector addition in the X-Y plane. 
Although inaccuracies in direction or distance measurements can accumulate over time, the 
surrogate 106 will only need to backtrack a short distance. 

To make distance measurements, the wheels 200 (or equivalently the wheel motors) 
of the surrogate 106 can have position encoders from which distance measurements can be 

10 obtained by odometry. While errors can be introduced in the distance measurement, 
backtracking will often cancel out these errors. These errors are introduced for example by 
compliance of the wheels 200 changing effective wheel radius, wheel slippage, and travel 
over uneven surfaces. However, it has been discovered since only relatively small 
backtracking distances are involved, the errors are not significantly compounded. 

15 Directional measurements may be made using a number of different methods. One 

method is to measure direction by means of external signals. The surrogate 106 can contain 
an electromagnetic compass, which is used to read direction while the surrogate 106 is in 
motion and the directional changes logged by the surrogate's computer/transceiver system 
122. The log can then be used when backtracking is required. In outdoor environments, the 

20 surrogate 106 can contain a satellite-signal-based global positioning system (GPS) receiver, 
which can also be used to provide the direction of motion. The drawback of electronic 
compasses is that they can sometimes take several seconds to settle on an accurate heading, 
and the drawback of GPS receivers is that they do not work indoors. 

A second method involves determining direction using internal measiwements. 

25 Gyroscopic devices can be used to determine the relative direction of motion of the surrogate 
106. However, gyroscopic devices are expensive. 

A third method of determining direction is by using odometry. Odometry measures 
distance by wheel rotation, and it can also be used to measure relative direction. Different 
methods are used based on the robotic drive system employed by the surrogate 106. 

30 In one embodiment of the surrogate 106 described above, a holonomic platform with 

synchronous drive is used. A holonomic platform can move in any direction without 
requiring turning of the platform. For example, the six wheels 200 of FIG. 2 effectively drive 
and steer synchronously. The direction can be inferred from encoders on the steering shaft or 
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motor. For example, steering and drive motors 320 and 322 are respectively provided with 
direction and distance encoders 324 and 326. 

Referring now to FIG. 4, therein is shown a differential drive system 400 in 
accordance with another embodiment of the present invention. The differential drive system 
5 400 both drives and steers with left and right wheels 402 and 404, respectively, mounted on 
two sides of the surrogate 106. The differential drive system 400 is not holonomic and the 
platform is tumed to steer. For added stability, the differential drive system 400 has 
additional small non-driven caster wheels 406. 

By mounting the left and right wheels 402 and 404 at the center of the sides of the 

10 surrogate 106, the surrogate 106 can turn in place by controlling the direction of the wheels 
so that one wheel rotates clockwise and the other counterclockwise. 

In the differential drive system 400, the direction is a function of the relative motion 
of the left and right wheels 402 and 404. For example, if the left wheel 402 is rotating 
forward slower than the right wheel, the surrogate 106 is rotating to the left (i.e., 

15 counterclockwise as seen from above in a circular direction 410) as well as moving forward 
in a direction 408. More precisely, the rate of tum is a function of the difference in wheel 
rotation speeds, so when the surrogate 106 is turning in a circle 406 to the left, the left wheel 
402 will be stationary and the right wheel 404 will be turning. 

When one wheel has tumed a distance on the ground equal to idl times the spacing R 

20 between the left and right wheels 402 and 404, the surrogate 106 will have rotated 90 
degrees. The spacing between the left and right wheels 402 and 404 is measured from the 
point where one wheel touches the ground to the same point on the other wheel. More 
generally, during a time period, if the right wheel 404 has traveled distance D further than the 
left wheel 402, then the surrogate 106 has turned left (counterclockwise as seen from above) 

25 by (7r*R*180)/D degrees. If the wheels contact the ground over a large area (e.g. tank-type 
treads), uncertainty is added to the computation due to variations in wheel slippage. 

It has been discovered that if directions are computed often enough (e.g., at 100 ms 
intervals), the average direction between one measurement and the next can be assumed to be 
valid for the whole time period between measurements without incurring significant error. 

30 In the differential drive system 400, the distance traveled is equal to the average travel 

of both left and right wheels 402 and 404. Thus, when the differential drive system 400 turns 
in place, the distance traveled will be zero. 
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Referring now to FIG. 5, therein is shown an example of a surrogate forward motion 
path 500. To simplify the FIGs. and discussion, it is assumed that the position information is 
logged once every second. In practice, a logging frequency of 10 times per second would be 
more conmion.) For purposes of illustration, the surrogate forward motion path 500 is 
5 shown as line segments 501 through 510, each of which is represented by an arrow indicating 
one second of surrogate movement. The surrogate 106 is assumed to have determined that 
the wireless signal has been lost at the end of the tenth second 510 of surrogate movement at 
a point 512. 

Referring now to FIG. 6, therein is shown an approximation of a point 600 that the 

10 surrogate 106 can be expected to reach moving at its pre-specified backtracking speed S in 
the next time interval T after initiating backtracking. In reality, the surrogate 106 cannot 
reach all points on a circle 602 in equal time, since it cannot change the direction of its 
motion quickly. To be more accurate, an oval could be used for the possible backtracking 
distance of the surrogate 106 instead of the circle 602. However, the approximation works 

15 since the curvature of the forward path is also limited. 

Once the surrogate 106 has entered backtracking mode, it navigates along its recent 
path in the reverse direction. For example, the surrogate 106 could reverse the last 10 
seconds of its motion. During backtracking, it is best if the surrogate 106 does not travel at 
high speed. Therefore, backtracking may take longer than the original forward motion. 

20 However, backtracking should not take too long, because any undue delay increases the 
amount of time that the user 124 is left incommunicado. 

During backtracking, the surrogate's computer/transceiver system 122 of FIG. lA will 
compute its expected position every sampling interval T (e.g., 100 ms) and compute a 
direction for the next time period. Given a pre-specified backtracking speed S (e.g., 0.5 ft/sec 

25 or 0.15 meters/sec), the surrogate's computer/transceiver system 122 can find a point on the 
original forward path that is a distance D = S/T away from its current position. This point is a 
point that the surrogate 106 can be expected to reach traveling at its pre-specified 
backtracking speed S in the next time interval T. 

Thus, a waypoint for the end of a backtracking interval will be on the intersection of a 

30 horizontal circle, such as the horizontal ckcle 602, and the line segments of the forward path, 
such as the surrogate forward motion path 500. Such a waypoint would be a waypoint 604. 

Once the surrogate 106 has been backtracking for several time intervals T, the D 
radius horizontal circle 602 will intersect the surrogate forward motion path 500 at two 
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points. When there is more than one intersection, the intersection with the oldest timestamp 
(as computed via interpolation from its endpoints) is chosen as the next waypoint. If the 
surrogate 106 is backtracking near its original forward speed, then an intersecting segment 
should have one endpoint inside the circle and one outside the circle. This means that the 
5 distance from die present surrogate position to the one endpoint will be greater than or equal 
to D, and the distance to the other segment endpoint will be less than or equal to D. 

However, if the surrogate 106 is moving significantly slower during backtracking than 
during the original forward motion, the circle may intersect a single segment in more than 
one place. In this case, all segment endpoints will be at a distance greater than D from the 
10 current surrogate position. To handle a general case, each line segment is tested for 
intersection with the circle, and the intersection with the oldest timestamp is chosen. 

In addition to autonomous backtracking, the surrogate 106 is also capable of 
autonomous forward navigation, in situations where the user 124 cannot find an alternative 
path with better wireless coverage or the user 124 desires to travel along a known or 
15 previously determined route to a specified destination at the remote location. 

Referring now to FIG. 7, therein is shown a method 700 of mobile telepresencing 
including: a block 702 of providing a surrogate moved under wireless control; and a block 
704 of moving the surrogate in reverse when the wireleiss control is lost. 

While the invention has been described in conjunction with a specific best mode, it is 
20 to be understood that many alternatives, modifications, and variations will be apparent to 
those skilled in the art in light of the aforegoing description. Accordingly, it is intended to 
embrace all such altematives, modifications, and variations which fall within the spirit and 
scope of the included claims. All matters hither-to-fore set forth herein or shown in the 
accompanying drawings are to be interpreted in an illustrative and non-limiting sense. 

25 
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